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IiyiAGE ANALYZING METHOD FOR DETECTING SIGNIFICANT CHANGES 



IN A TIME SEQUENCE OF IMAGES 



1 . Field of the Invention 

This invention relates to the analysis of images and 
more particularly to the analysis of images in a sequence 
of images and in a region of interest. 

2 . Description of the Prior Art 

There are many applications where it is desirable to 
analyze an image to determine if there has been a change 

10 to that image. Examples of such applications are 
surveillance, industrial or manufacturing processes. One 
example of such a process is the making of paper. 

There are systems for monitoring a paper web that 
use multiple cameras. For example in the Smart Advisor 
system sold by ABB Automation Inc. a papermaking machine 
may have cameras stationed at various locations along the 
paper web. If the paper web should break, the 

papermaking machine operator can look at the sequence of 
images captured by each of the cameras for the time 

20 period just prior to the tear in order to try to find the 
specific image that shows the reason for the tear. The 
sequence of images from the cameras are synchronized with 
each other so that each sequence of images shows the same 
section of the paper web. 

Using the present system the operator cannot 
determine if the paper web has localized defects such as 
holes, discoloration, edge flips and changes in takeoff 
angle unless these defects cause the web to tear. The 
detection of localized defects when they occur will 

30 increase the quality of the paper produced by the 
papermaking machine. Thus it is desirable to make the 
detection of the cause of tears in the paper automatic 
and also to automatically detect upon occurrence 
localized defects that do not give rise to tears in the 
web . 



There are cameras such as the Panasonic WV-CL830/WV- 
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CL834 color CCTV camera, which are typically used in 
surveillance systems, that have the ability to set up a 
grid that is divided into cells. The camera has the 
capability to monitor the cells to see if there has been 
a change to the viewed image, for example, the brightness 
has changed, and if so send an alarm signal to the 
surveillance system. Upon receipt of the alarm signal an 
individual must view the scene to determine why the image 
changed and if that change is significant. 

10 U.S. Patent No. 5,245,424 describes a defect 

defection method. In the method described therein, a 
video camera picks up an image of a light irradiated 
object. Two inspection zones adjacent each other are 
established on the picture screen. The zones each have a 
preset size and the brightness of each of the two zones 
is sensed and accumulated as the inspection zones are 
indexed across the screen. When the difference between 
the accumulated values of the respective inspection zones 
exceeds a preset threshold a defect is detected. 

20 While the image analysis techniques of the prior art 

do allow a change in an image to be detected they do not 
include a user designated zone which will not have 
interesting changes therein but can be used to assess the 
image quality. Further none of the prior art techniques 
provide for the automatic assignment of cells into a zone 
which may include interesting changes and a zone of the 
image that does not have to be analyzed. Still further 
none of the prior art image analysis techniques allow for 
the analysis of the image cells to be carried out on 

30 compressed data. Yet still further none of the prior art 
image analysis techniques remove normal periodic 
variations from the image prior to computation of the 
feature statistics. The image analysis technique of the 
present invention has these properties. 

Summary of the Invention 
A method for analyzing an image. The method has the 
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steps of: 

(a) dividing a reference image into: 

(i) one or more detection cells collectively 
comprising a detection zone in which a 
statistically significant change in one 
or more predetermined features in the 
reference image is expected to occur; 

(ii) one or more veto cells collectively 
comprising a veto zone in which any 

10 detection in the veto zone of a 

statistically significant change in one 
or more predetermined features in the 
reference image is used to disable any 
detection of a statistically significant 
change in one or more predetermined 
features in the reference image in the 
detection zone; and 

(iii) zero or more ignore cells collectively 
comprising an ignore zone in which no 

20 computation is performed. 

A method for analyzing an image. The method has the 
steps of: 

(a) using a reference image to position analysis 
cells in the image; and 

(b) dividing the reference image into one or more 
types of analysis cells comprising: 

(i) one or more detection cells collectively 
comprising a detection zone in which a 
statistically significant change in one 

30 or more predetermined features in the 

reference image is expected to occur; 

(ii) one or more veto cells collectively 
comprising a veto zone in which any 
detection in the veto zone of a 
statistically significant change in one 
or more predetermined features in the 
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reference image is used to disable any 
detection of a statistically significant 
change in one or more predetermined 
features in the reference image in the 
detection zone; and 
(iii) zero or more ignore cells collectively 
comprising an ignore zone in which no 
computation is performed. 
A method for analyzing an image. The method has 
the steps of: 

(a) using a reference image to position analysis 
cells in the image; and 

(b) dividing the reference image into one or more 
zones, each of the zones comprising a collection of cells 
of the same type, the one or more zones comprising: 

(i) a detection zone comprising one or more 
detection cells in which a statistically 
significant change in one or more 
predetermined features in the reference 
image is expected to occur; 

(ii) a veto zone comprising one or more veto 
cells in which any detection in the veto 
zone of a statistically significant 
change in one or more predetermined 
features in the reference image is used 
to disable any detection of a 
statistically significant change in one 
or more predetermined features in the 
reference image in the detection zone; 
and 

(iii) an ignore zone comprising zero or more 
ignore cells in which no computation is 
performed. 

A method for analyzing an image. The method has the 
steps of: 

(a) dividing a reference image into one or more 
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detection cells collectively comprising a detection zone 
in which a statistically significant change in one or 
more predetermined features in the reference image is 
expected to occur; 

(b) digitizing the divided reference image into an 
array of pixels having a predetermined format; 

(c) identifying all of the pixels associated with 
each of the one or more detection cells other than pixels 
associated with an ignore cell contained within a 

10 detection cell; 

(d) calculating from the amplitude of all of the 
pixels identified with each of the one or more detection 
cells values for one or more features for each of the one 
or more detection cells; 

(e) comparing the values calculated for the one or 
more features for each of the one or more detection cells 
with a predetermined criteria for each of the one of more 
features and identifying each of the one or more 
detection cells in which the calculated value exceeds the 

20 predetermined criteria for each of the one or more 
features; 

(f) determining the total number of the identified 
one or more detection cells; and 

(g) responsive to the total number exceeding a 
predetermined threshold, setting a zone status to a 
predetermined status . 

Description of the Drawing 
Fig, 1 shows a block diagram of the basic hardware 
components required to implement the method of the present 
30 invention. 

Figure 2 is a schematic of an image prior to the 
cell assignments. 

Figure 3 is a schematic of an image with cell 
assignments . 

Figure 4 is a flowchart of the cell detection logic. 
Figure 5 is a flowchart of the zone detection logic. 
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Figure 6 is a flowchart of the image detection 
logic. 

Figure 7 is a flowchart of the logic used for 
setting a cell to be an Ignore cell. 

Figure 8 is a flowchart of the algorithm for 
filtering dominant periodic components from the feature 
history. 

Description of the Preferred Embodiment (s) 
Referring now to Figure 1 there are shown the basic 

10 hardware components required to implement the present 
invention. A video camera 10 is used to image the scene. 
The video signal is digitized by digitizer 12 into a two 
dimensional array of pixels. Each pixel has an amplitude 
characterized by the amount of red, green, and blue (r, 
g, b) color at the pixel location. Although the following 
discussion uses the RGB format, it is understood that the 
method can easily be applied to other image 
characterizations such as gray scale or other color 
models such as YUV. In the YUV model, the Y component 

20 represents the intensity of the pixel and the U and V 
components represent the chromaticity . 

The image is analyzed by processor 14 using the 
method described below and, when an image is identified, 
the human interface 16 or another machine interface is 
notified so that the image can be saved or marked and 
further action taken. 

The identification of significant regions in the 
image is an important feature of the method of the present 
invention. Consider a schematic image such as the image 

30 20 shown in Figure 2 which has therein representative 
content 22, 24 and 25. While the representative content 
22, 24 and 26 is shown in image 20 as geometric shapes it 
should be appreciated that the representative content can 
be any content, for example, a pastoral scene, a high 
speed packaging line or an industrial process such as the 
making of paper. 
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Referring now to Figure 3, the image 28 is a 
reference image used to position analysis cells in the 
image as will be described in more detail below. As part 
of the setup procedure, the user of the method of the 
present invention divides a reference image based on its 
visual content into cells consisting of one or more 
Detection cells, such as cells 30, 32, 34 and 36, labeled 
in Fig. 3 with a 'M", in which an interesting change is 
expected to occur in the image, zero or more Ignore 

10 cells, such as cell 38, labeled in Fig. 3 with an ^^i", in 
which nothing of interest will occur, and zero or more 
Veto cells, such as cell 40, labeled in Fig. 3 with a 
''v", in which any detection is used to disable any 
detection in the detection zone. 

As used herein a zone is a collection of possibly 
connected or disconnected cells of the same type. 
Therefore, at the end of the setup procedure the 
reference image has one detection zone, one veto zone and 
one ignore zone and the cells in each zone may or may not 

20 be connected to each other. 

Only statistically significant changes in the 
Detection cells can identify an image as interesting. The 
Ignore cells are areas of the image that are not 
analyzed. Any pixel, for example pixel 37 shown in Fig. 
3, which is outside of an identified cell, that is, a 
Detection cell or a Veto cell, is by default in an Ignore 
cell . 

A change to the image that is not of interest may 
also trigger a change in a Detection cell. The Veto 
30 cells are used to identify changes to the image that are 
not of interest and ensure that the method of the present 
invention ignores those changes. Examples of changes to 
the image that are not of interest include a loss or 
degradation of the video signal. A Veto cell would 
generally be located in an area of the image in which no 
change is expected. 
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The Detection, Veto and Ignore cells can be of 
arbitrary size, shape, location, and number. As is shown 
in Fig. 3, an Ignore cell can be inside of a Detection 
cell. While not shown in Fig. 3, a Veto cell can be 
inside of a Detection cell or vice versa. 

As was described above, one application of the 
method of the present invention, by way of example and 
not by way of limitation, may be in a papermaking 
machine. The reference image in that application would 

10 usually be the paper web and surrounding parts of the 
machine. In the setup procedure the operator of the 
papermaking machine would typically select all of the 
cells in the paper web as the detection zone and the 
cells off sheet as the veto zone. 

Figure 4 shows a description of the analysis 
performed on each Detection and Veto cell by the cell 
detection logic 41. The digitized image array 42 is 
transferred from the digitizer 12 to the analyzer 14. 
Let the digitized image array 42 be described by the 

20 vector (rij, gij, bij) where i ranges over the columns in 
the array and j over the rows and rij represents the 
amount of red color, gij represents the amount of green 
color, and bij represents the amount of blue color at 
pixel i j . 

The identification of the pixels in the cell 
performed by block 44 is based on all the pixels in the 
identified cell region. Let the notation "ij in Ck" 
denote the set of pixels ij in cell Ck. If a Detect or 
Veto cell contains an Ignore cell, then the pixels in the 
30 Ignore cell are not included in the pixels in the Detect 
or Veto cell. 

For a cell Ck/ consisting of N pixels, a set of 
features or discriminants are computed 4 6 based on the 
pixel amplitudes in the cell. Possible features can be 
based on the intensity of each pixel in the cell: 



Pixel intensity. 
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Average cell intensity, / = — 

^ ' /,y in Cj^ 

Intensity variance, ^/ =— ^{^ij~^J 

^ iJ in Ck 



Other features can be based on the colors in the cell. 

Z^i/' Hsij^ H^ij 



Average color, c= — 



in Cjt ij in C^^ ij in Q J 

The method requires a training period so that the 
feature statistics can converge to reliable values and a 
decision block 48 determines if the training is 
^11 completed. During the training period no cell alarm 

logic is applied and only the feature statistics are 
I^L 10 updated in block 50. The training period consists of a 
H sequence of M images In with the cell feature computation 

Si applied to each cell of each image. The user of the 

3 method of the present invention defines either the time 

ri^ duration of the training period or the number of images, 

y that is, M. 

Let fnmki be the value of feature 1 for cell k of zone 
m in image n with m = Detect or Veto. For convenience of 
notation, the zone, cell, and feature subscripts will be 
dropped in the discussion below and only the image index 
20 is retained. During the training period the mean and 
variance of each feature is computed. 

Feature mean, / = ^ y\fm 



Feature variance, V ^{fm-fj 

^ m=\M 

Once the training period is complete, the feature 
statistics continue to be maintained so that the method 
can adapt to slow changes in the statistics. For each 
image, zone, cell, and feature a running mean and 
variance is maintained. 
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History feature mean, Jnmki-^ l^jmi 

History feature variance, Kmki=~j Y.[finki^fnmki) 

In addition to the linear statistical average 
described above, other averaging techniques such as an 
exponential average could be employed for computing the 
historical feature means and variances. 

As each image is analyzed once the training period 
is over, the feature values are compared in decision 
block 52 to the statistical values and the sensitivity 
10 value aki for cell k and feature 1. If for any feature 1 
in cell k of zone m in image n, 

{fnmkl ~ fn-\,m,kj ) ^ ^kl -yj^n-lm.kj 

then cell status Snmk = 1^ otherwise it is 0. If the cell 
status is 0, then the feature statistics are updated in 
block 50 as described above. The cell status is set to 1 
or 0 in blocks 54 and 56, respectively. A cell status of 
1 is a preliminary indication of the detection of a 
statistically significant change in that cell. 

Once the cell logic has been applied to every cell Ck 
20 in zone m (m=Detect or Veto) , the zone logic 60 of Figure 
5 is applied to those cells. The number of cells with a 
true status, that is, a cell of status of 1, is 
determined in block 62. If decision block 54 determines 
that the count of the number of cells with a cell status 
of 1, 

z = V 9 

«/« / ^ nmk ' 
k 

is greater than threshold pm then block 66 sets the zone 
status Tnm = If otherwise block 68 sets the zone status to 
0. If the zone status is 1, then the image capture time, 
30 tnm/ is recorded. The threshold pm is set by the user of 
the method of the present invention. 

The alarm logic 70 for the image is shown in Figure 
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6. If decision block 72 determines that the Detect zone 
status Tn, Detect = 0, then the image status Un is set to 0 
in block 74 and no significant change is indicated. If 
block 72 determines that Tn, Detect = 1, then the veto 
period, Xf of block 7 6 must pass allowing the computation 
of the veto zone status of future images to be computed. 

The image status Un is 1 (block 79) if in loop 78, 
for all images i with veto zone status ti,veto = 1 (block 
78a) , there are no veto zone times ti,veto (decision block 

10 78c) such that abs ( tm^oetect - ti,veto ) < T (decision block 
78b), otherwise the image status is 0. An image status 
of 1 indicates that a statistically significant change 
has occurred in the image that is not associated with any 
changes in the Veto Zone. This image can then be viewed 
from the user interface or marked for special use by 
another program or process. 

An enhancement to the cell assignment procedure 
discussed above is shown in the logic 80 of Figure 7. 
This technique allows the automatic assignment of cells 

20 into either Detect or Ignore zones. The method uses a 
training set of images of number M with each image 
divided into a grid pattern of cells consisting of K 
columns and L rows to identify in block 82 the pixels in 
a cell. For example if the image is 640 x 480 pixels, 
and a cell grid of 20 columns and 20 rows is defined, 
then each cell is 32 x 24 pixels. Based on the feature 
statistics, each cell is classified as either a Detect 
cell (block 92) or an Ignore Cell (block 94) . Veto cells 
are not assigned by this technique. 

30 Logic 80 then proceeds to block 84 which has the 

same function as block 46 described in connection with 
Fig. 4. Thus block 84 computes the features in a cell 
based on the pixel amplitude in the cell. 

Logic 80 then proceeds to block 88 where it computes 
the feature statistics for each cell. The computation of 
the feature statistics for each cell is performed in the 
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same manner as described above in connection with the 
training period of Figure 4. 

History feature mean, fki =T7^fiki ^ 

where fi^i is the feature 1 value cell k in image i 

History feature variance, T7 SUjw "/aJ 

Thresholding logic in the form of decision block 90 
is then applied to the mean and variance to determine the 
cell type. Let ai and ah be the low and high threshold 
limits for the mean and pi and ph the low and high 
10 threshold limits for. the variance. The threshold limits 
are set by the user of the method of the present 
invention . 

Block 90 determines that a cell is an Ignore cell if 
7^ < ai or 7^ > ah or V^^ < pi or > ph . 

otherwise the cell is a Detect cell. If the feature is 
the intensity in the cell, then the cell is an Ignore 
cell if it is too dark or too bright, or if the variance 
is too small or too large. The logic 80 is appropriate 
in cases where the method is tuned to detect activity in 

20 areas that are generally inactive. 

An additional variation on the technique can be 
utilized when a compression algorithm is applied to the 
image prior to analysis. In many cases the video signal 
is both digitized and compressed by special hardv/are. 
The discussion above of the analysis method of Figure 7 
assumes that the image array is uncompressed. A total 
decompression of the image is very time consuming and can 
limit the real-time rate of the analysis method. 
However, some cell features can be computed with only a 

30 partial decompression. Many compression algorithms such 
as the MJPEG and MPEG standards process the image by 
applying the Discrete Cosine Transform (DCT) to blocks of 
pixels. If the cell positions discussed above are 
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constrained to correspond to the boundaries of the DCT 
blocks, then the feature computation can be optimized. 

As an example, consider the MJPEG compression 
algorithm. Although it is beyond the scope of the 
description of the present invention to provide a 
detailed description of the algorithm, a brief overview 
is given herein. The MJPEG compression algorithm first 
converts the image from the RGB model to the YUV model. 
As described above, the Y component represents the 
10 intensity of the pixel and the U and V represent the 
chromaticity . The image pixels for each component are 
then divided into 8x8 blocks and the Discrete Cosine 
Transform is applied to each block. The resulting block 
DCT coefficients are quantized and then run length 
encoding and Huffman encoding are applied to the 
coefficients . 

In the case where the feature is the intensity of 
the cell and the cell is composed of a number of DCT 
blocks, the average intensity of the cell can be computed 
20 by decompressing only the DC term of the included DCT 
blocks and averaging these values. This represents a 
significant savings in the feature computation time for 
the cell. 

The described MJPEG compression analysis algorithm 
is designed to identify images with feature values in a 
selected region of the image with statistically 
significant deviation from the norm. Many scenes have a 
normal periodic variation that may generate a cyclical 
variation in the feature statistics. The sensitivity of 
30 the technique can be improved if the periodic variation 
is removed prior to computation of the feature 
statistics. This removal of the periodic variation is 
accomplished by the logic 100 of Figure 8. 

As described above, the feature statistics are 
computed in block 102 from a sequence of M feature 
values. Let fnmki be the feature 1 value of cell k in zone 
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m of image n where n spans the M images. To simplify the 
notation below, only the n index will be retained. A 
frequency analysis can be applied to this sequence to 
identify and remove any dominant periodic variation. The 
discrete Fourier transform of the sequence of features, 
computed in block 104, is given by: 

Fiw)= Y.f>y^u\ where W^=e<'"l''^ 

The fractional amplitude of any particular frequency 
component is: 

1 0 r{w) = ^F(w)'F{w) l \ Y^Fiiypii) 

In block 106 all components that have a fractional 
amplitude exceeding a predetermined threshold y are 
identified and let the set S be the set of indices 
satisfying this criterion. 

S = set of w such thatr(>v)>/ 
The components in S are then removed by block 108 
from the feature sequence by applying the inverse 

transform resulting in a filtered feature sequence, /^^^j^, . 

fnmkl ~ fnmkl ~ TT ^ ^(^^A/ 

20 The features statistics and thresholding are 

replaced in block 110 with those based on the filtered 
feature values. 

1 - 

History feature mean, fnmki=— l^f^i 

History feature variance, Kmki=^ Y^finki -fnmu) 
If for any feature 1 in cell k of zone m in image n, 

(fnn.ki - fn-i,.,kj)> ^kl^|K-ln.^J then Cell status Snmk = 1, otherwise 
it is 0. 

It is to be understood that the description of the 
preferred embodiment (s) is (are) intended to be only 
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illustrative, rather than exhaustive, of the present 
invention. Those of ordinary skill will be able to make 
certain additions, deletions, and/or modifications to the 
embodiment (s) of the disclosed subject matter without 
departing from the spirit of the invention or its scope, 
as defined by the appended claims. 



